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PREFACE 


This  is  the  report  of  the  Panel  on  Assessment  of  Computer  Science 
Research  Activities  of  the  Office  of  Naval  Research  (ONR).  The 
Panel  was  formed  under  the  auspices  of  the  Naval  Studies  Board  of 
the  Commission  on  Physical  Sciences,  Mathematics,  and  Resources  of 
the  National  Research  Council  (NRC). 

The  ONR  is  responsible  for  all  the  basic  research  conducted  by 
the  Navy.  A  portion  of  ONR's  funds  is  earmarked  for  computer 
science  (CS)  research.  A  portion  of  the  CS  funds  is  administered  by 
the  Contract  Research  Program  (CRP)  of  ONR.  The  Panel  was  requested 
to  evaluate  the  research  program  of  the  CRP  and  also  related  basic 
research  on  computer  science  being  performed  elsewhere  in  the  Navy 
(e.g.,  the  Naval  Research  Laboratory,  the  Naval  Air  Systems  Command, 
NAVSEA,  and  the  Space  and  Naval  Warfare  Systems  command). 
Representatives  of  the  CRP  and  the  various  Naval  Commands  who 
receive  computer  science  research  funds  from  ONR  made  presentations 
to  the  Panel.  (The  agenda  is  attached  as  Appendix  A.)  The  Panel 
evaluated  the  information  presented  and  offers  its  comments  in  this 
report.  some  of  the  Panel's  comments  are  directed  to  the  ONR  in 
general,  including  all  claimants,  while  other  comments  are  directed 
to  CRP  in  particular. 

The  Panel  was  originally  requested  to  present  its  evaluation  in 
a  letter  report  to  ONR.  In  early  1986,  however,  the  NRC  requested 
that  the  Panel's  brief  report  be  presented  in  a  normal  report  format 
while  also  undergoing  the  regular  NRC  review  process.  Because  the 
funds  provided  under  the  original  charge  were  not  sufficient  for  a 
series  of  meetings,  the  Panel  did  not  have  the  opportunity  to  review 
in  depth  all  current  ONR  projects,  to  investigate  extensively  the 
Navy's  needs  in  computing,  to  study  at  length  the  internal 
procedures  of  the  CRP,  or  to  widen  the  audience  of  the  report. 

,  Consequently ,  the  Panel's  comments  on  these  matters  are  limited  to 
[general  observations  directed  to  ONR  program  managers. 

Most  of  the  comments  contained  in  this  report  are  derived  from 
the  information  presented  to  the  Panel  at  its  meeting  in  October 
1985.  Changes  in  the  Navy's  priorities  or  methods  of  doing  business 
since  that  date  may  not  be  reflected  here.  The  Panel  had  no  access 
to  classified  information;  in  some  areas,  the  Navy  may  actually  be 
using  computing  technology  of  different  technical  sophistication 
than  was  shown. 

The  Panel  was  generally  pleased  with  the  way  in  which  the  Navy 
has  conducted  its  computer  science  program  and  believes  that  the 
program  has  made  many  good  investments  and  has  an  excellent  record 
of  successes.  The  Panel  has  a  few  minor  concerns,  which  will  be 
expressed  in  this  report  as  suggestions  for  improvements. 

On  behalf  of  the  Panel,  I  would  like  to  thank  the  members  of  the 
ONR  staff,  as  well  as  those  from  the  various  Naval  Commands  and 
Labor ator ies ,  who  participated  in  this  project,  as  well  as  the 
liaison  representatives  and  NRC  staff. 

Peter  J.  Denning,  Chairman 

Panel  for  the  Assessment  of  Computer  Science 

Activities  of  the  Office  of  Naval  Research 
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ABSTRACT 


A  Panel  of  the  Naval  Studies  Board  of  the  National  Research 
Council  met  for  two  days  in  October  198S  to  assess  the  computer 
science  programs  of  the  Office  of  Naval  Research  (ONR).  These 
programs  are  supported  by  the  contract  Research  Program  (CRP) 
as  well  as  the  Naval  Research  Laboratory  ( NRL ) ,  the  Naval  Air 
Systems  Command,  and  the  Space  and  Naval  Warfare  Systems 
Command.  Based  principally  on  information  presented  to  the 
Panel  at  the  two-day  meeting,  as  well  as  their  knowledge  of  the 
field,  the  panel  offers  comments  on  the  computer  science 
research  of  the  Navy  and  suggestions  for  further  improvement  cf 
a  program  that  has  already  made  numerous  good  investments  with 
a  number  of  the  best  projects  in  the  country  and  an  excellent 
record  of  success.  ONR,  for  example,  was  encouraged  to  take 
further  advantage  of  its  entrepreneurial  flexibility  to  improve 
technology  transfer  between  computer  science  and  the  Navy  by 
mechanisms  such  as  focused  meetings,  consulting,  and  scientific 
exchange  programs.-  Four  additional  mechanisms  for  obtaining 
advice  on  the  computer  science  program  and  project  evaluation 
were  suggested:  contractor's  workshops,  advisory  panels  for 
program  review  and  general  guidance,  more  use  of  standard  ONR 
formats  for  project  descriptions,  and  better  publicity  for  ONR 
pro jects .  ‘  ^The  Panel  also  identified  five  potential  new  areas 
requiring  substantial  basic  research:  real-time  systems, 
system  structure  concepts,  parallel  programming,  formalization 
of  expert  systems,  and  representation  of  physical  objects. 
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1.  SUMMARY 


At  the  request  of  the  Office  of  Naval  Research  ( ONR ) ,  a  Panel 
of  the  Naval  Studies  Board  (acting  on  behalf  of  the  then 
currently  inactive  Computer  Science  and  Technology  Board)  of 
the  National  Research  Council  was  convened  to  assess  the 
computer  science  programs  of  ONR.  This  program  embraces 
research  funded  by  the  Contract  Research  Program  (CRP)  and 
related  basic  research  being  funded  elsewhere  in  the  Navy  (for 
example,  the  Naval  Research  Laboratory,  the  Naval  Air  Systems 
Command,  and  the  Space  and  Naval  Warfare  Systems  Command).  The 
Panel  held  a  two-day  meeting  at  the  National  Academy  of 
Sciences  in  October  1985,  at  which  various  naval  research 
personnel  reported  on  their  programs  and  their  dispositions  of 
computer  science  research  monies.  (The  agenda  for  this  meeting 
is  attached  as  Appendix  A.)  Based  principally  on  the 
information  presented  to  the  Panel  at  that  meeting,  the  Panel 
concluded  that  CRP  program  managers  have  made  numerous  good 
choices:  they  have  sponsored  some  of  the  best  projects  in  the 

country  and  have  had  an  excellent  record  of  successes. 

In  this  report,  the  Panel  comments  on  the  ONR's  computer 
science  research  program  and  offers  a  few  suggestions  for 
improvement.  The  CRP  and  the  other  claimants  on  ONR  funds  were 
encouraged  to  take  further  advantage  of  their  entrepreneurial 
flexibility  to  improve  technology  transfer  between  computer 
science  and  the  Navy  by  mechanisms  such  as  focused  meetings, 
consulting,  and  scientific  exchange  programs.  While  not  able 
to  offer  a  technical  review  of  each  of  the  currently  funded 
projects  of  the  CRP  or  the  other  claimant  commands,  the  Panel 
noted  a  number  of  these  projects  whose  significance  has  already 
been  recognized  among  computer  scientists. 

Additional  mechanisms  for  obtaining  advice  on  the  computer 
science  program  and  the  evaluation  of  projects  were  suggested, 
notably,  semiannual  contractor's  workshops,  a  formal  advisory 
panel  for  program  review  and  general  guidance,  and  more 
publicity  of  research  results.  The  Panel  also  recommended  more 
prominent  use  of  the  ONR  standard  format  for  describing 
projects . 

Finally,  the  Panel  identified  five  potential  new  areas  of 
investigation  requiring  substantial  basic  research,  viz., 
real-time  systems,  system  structure  concepts,  parallel 
programming,  formalization  of  expert  systems,  and 
representation  of  physical  objects,  and  offered  brief  comments 
on  each. 
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2.  BACKGROUND  ON  COMPUTER  SCIENCE 
2.1.  The  Discipline 

The  Panel  understands  that  ONR  senior  management  sees  computer 
science  as  a  young  discipline  that  is  not  yet  fully  established 
as  are  the  other,  more  traditional  disciplines.  To  command  the 
support  from  ONR  that  it  requires  and  deserves,  computer 
science  must  be  shown  to  have  firm  scientific  roots,  and  it 
must  also  be  shown  to  be  important  to  the  Navy.  The  following 
subsections  are  brief  summaries  of  the  evidence  for  these  two 
claims . 

Computer  science  is  the  systematic  study  of  processes  that 
transform  information:  principles  underlying  their  analysis, 
implementation,  efficiency,  design,  and  application.  The 
fundamental  question  underlying  all  of  computer  science  is 
"What  can  be  automated? " ^  This  discipline  was  born  in  the 
mid-1940s  with  the  invention  of  the  s tored-pr ogram  electronic 
computer  and  has  growr  rapidly  ever  since.  The  roots  of 
computer  science  can  ue  traced  back  several  centuries  in 
mathematics,  engineering,  and  logic. 

Computer  science  can  be  viewed  as  a  discipline  of  eleven 
subfields.  Each  subfield  addresses  fundamental  questions  and 
has  made  significant  intellectual  and  practical 
accomplishments.  The  subfields  of  computer  science  are2 
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Programming  languages  and  methodology 
Algorithms  and  data  structures 
Architecture  (hardware) 

Theory 

Numerical  computation 

Database  and  information  retrieval  systems 
Human  interface 

Parallel  and  distributed  computation 
Operating  systems 
Dependable  computing 
Artificial  intelligence 

The  subfields  are  by  no  means  mutually  exclusive.  Each  has 
its  own  theoretical  component;  most  have  devised  specialized 
programming  languages  as  notation  for  algorithms  and  data 
structures;  most  implementations  are  on  machines  with  operating 
systems  connected  to  networks;  most  deal  with  problems  having 
components  that  can  execute  in  parallel. 

Methodologies  for  design  and  analysis  pervade  all 
subfields.  In  computer  science,  these  methodologies  are  not 
treated  as  separate  subject  areas.  For  example,  software 
engineering  is  a  design  methodology  that  embraces  all  eleven 
areas. 
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2.2.  The  Navy's  Interest 

Computer  science  is  a  critical  discipline  to  the  Navy.  The 
E-2C  Airborne  Early  V.arning  aircraft,  the  Aegis  Anti-Air 
'Warfare  systems,  and  signal-processing  methods  used  in  passive 
sonar  arrays  illustrate  the  importance  of  computing  in  today's 
air,  surface,  and  submarine  naval  operations. 

Computing  will  play  an  even  more  important  role  in  future 
military  systems.  Computerizing  algorithmic  tasks  that  are 
currently  performed  by  humans  can  increase  system  performance, 
overcome  personnel  shortages,  and  compensate  for  the  lower  10 
levels  of  an  all -volunteer  Navy.  Computer izat ion  can  reduce 
manpower  requirements  as  well  as  improve  the  safety,  logistics, 
financial,  and  weight  problems  that  accompany  large  crews.  The 
U.S.  Navy  expects  to  use  the  nation's  computing 
expertise — which  far  surpasses  that  of  other  nations--as  a 
"force  multiplier"  that  counters  numerical  aisaovantage  with 
superior  technology. 

Some  of  the  Navy's  specific  needs  in  improved  computing 
include  real-time  systems,  command  and  control  systems, 
fault-tolerant  distributed  computing,  innovative  concepts  for 
system  structuring,  extensions  of  software  engineering  to 
parallel  and  distributee  systems,  reliable  and  predictable 
expert  systems,  automated  real-time  logistics  planning  systems, 
and  automated  mechanisms  for  manufacturing  replacement  parts 
for  naval  equipment.  In  a  later  section  of  this  report,  the 
Panel  recommends  greater  emphasis  on  some  of  these  areas. 

It  may  be  tempting  to  conclude  that  engineering,  not 
research,  is  the  best  path  to  meeting  the  Navy's  pressing  needs 
in  automat ion--i . e . ,  that  a  greater  percentage  of  tunas  needs 
to  be  allocated  for  applied  research  and  development 
(categories  6.2  and  higher)  than  for  basic  research  (category 
6.1).  The  Panel  believes  that  this  conclusion  would  be  wrong 
and  detrimental  to  the  Navy's  interest.  For  example, 
engineering  technology  is  proaucing  new  types  of  sensors, 
front-end  processors,  control  mechanisms,  and  networks  at  a 
much  higher  pace  than  the  research  community  is  discovering 
principles  for  integrating  these  components  into  reliable, 
fault-tolerant,  real-time  systems.  Software  engineering  is  not 
producing  concepts  for  programming  parallel  and  distributed 
systems  at  the  same  pace  that  processors  are  being  lashed 
together  into  arrays  or  computers  into  distributed  systems. 
Theories  of  knowledge  representation  are  insufficiently 
advanced  to  make  expert  systems  reliable  or  their  performance 
in  untested  situations  predictable.  The  architecture  concepts 
needed  tor  real-time  systems,  pr og r amming  concepts  for  parallel 
and  distributed  systems,  dependability  concepts  for  large 
distributed  systems,  and  knowleage  representation  concepts 
needed  for  expert  systems  are  not  being  studied  by  the 
engineers  who  build  these  systemis.  They  are  being  studiea  by 
scientists,  but  to  a  much  lesser  extent  than  is  needed  to 
discover  solutions  to  the  problem.  The  Navy  needs  to  give  more 
eiriDhasis  to  basic  research  in  these  areas. 


There  is  quantitative  support  for  the  claim  that  basic 
research  has  as  much  to  contribute  to  the  advancement  of 
computing  as  has  engineering.  In  1984,  Bentley  reported  a 
study  by  Rice  about  the  improvements  in  the  performance  of 
numerical  codes  since  1945  .  ^  Over  that  period,  engineering 
advances,  as  measured  by  speeds  of  the  fastest  computers,  were 
responsible  for  a  factor  of  10^  speedup.  Over  the  same 
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3.  TECHNOLOGY  TRANSFER 

Based  on  comments  made  by  various  persons  making  presentations, 
the  Panel  perceives  that  technology  transfer  is  important  to 
the  Navy.  By  this  is  meant  the  bidirectional  flow  of 
information,  results,  and  know-how  between  the  fleet  and  the 
computer  science  community.  Close  interaction  between  Navy 
personnel  and  computer  science  researchers  will  help  to  speed 
the  introduction  of  leading-edge  computing  technology  into 
naval  systems  and  operations.  Some  of  the  large, 
distributed-systems  problems  facing  the  Navy  can  be  an 
important  stimulus  to  a  large  segment  of  computer  science 
researchers . 

The  information  presented  to  the  Panel  showed  a  wide 
variation  in  the  sophistication  of  computer  technology 
currently  deployed  in  the  fleet.  In  some  domains,  such  as 
signal  processing,  deployed  systems  are  advanced  far  beyond 
their  commercial  counterparts.  Other  domains  appear  to  lag  the 
civilian  state  of  the  art;  for  example,  the  Panel  was  told  of  a 
front-line  avionics  system  (not  the  A-7  System)  that  was  built 
using  hardware  and  software  methods  that  are  two  decades  old. 
Systems  lagging  civilian  technology  are,  in  the  Panel's 
opinion,  needlessly  expensive  and  offer  reduced  performance. 

(It  is  possible  that  the  Navy  has  more  advanced,  but 
classified,  computing  technologies  in  these  domains.) 

The  software  research  at  Naval  Research  Laboratories,  led 
by  Parnas,  and  motivated  by  the  A-7  Corsair  light  attack 
aircraft,  is  an  example  of  technology  transfer  at  its  best. 

The  project  has  led  to  important  new  techniques  in  software 
engineering  and  made  substantial  contributions  to  both  theory 
and  practice.  Some  of  the  results  are  in  current  use  in  the 
A-7  Avionics  System  in  the  Navy's  China  Lake  group,  for 
maintaining  the  aircraft's  avionics  software.  The  technology 
transfer  was  bidirectional;  the  real  problems  inherent  in  the 
system  led  Parnas  to  fundamental  insights  that  have  been 
disseminated  to  a  wide  community  of  software  engineers.  This 
project  involves  all  four  interested  groups:  academic 
researchers,  naval  research  laboratories,  development 
contractors,  and  the  fleet. 

The  Panel  appreciates  that  the  CRP  and  other  claimants  may 
be  impeded  from  reaching  their  goals  in  technology  transfer  by 
factors  beyond  the  scope  of  this  report,  for  example, 
procurement  policies  and  cost  structures  of  defense  contracts. 
Nonetheless,  the  Panel  would  like  to  encourage  especially  CRP 
but  also  the  other  claimants  to  take  additional  advantage  of 
their  entrepreneurial  flexibility  to  improve  technology 
transfer  between  computer  science  and  the  Navy.  Some  of  the 
more  promising  mechanisms  follow. 

Meetings --foe us  on  a  specific  technology  area;  seek 
attendance  by  a  mixture  of  people  from  among  universities, 
research  contractors,  naval  laboratories,  development 
contractors,  and  operational  naval  personnel;  keep  each  meeting 
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small  to  encourage  interactions;  arrange  for  the  proceedings  of 
these  meetings  to  be  published  and  distributed. 

Consulting — seek  personal  interactions  between 
qualified  computer  scientists  and  Naval  Commands. 

Exchange  Programs--invite  academics  to  visit  naval 
laboratories  or  contractor  facilities  while  on  sabbatical; 
permit  naval  scientists  to  spend  a  year  on  campus. 
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NAVY  FUNDING  OF  COMPUTER  SCIENCE  RESEARCH 


4  . 


4.1.  General  Picture 

Table  4.1  summarizes  basic  research  funds  (category  6,1)  for 
computer  science  in  the  major  agencies.  The  Navy  contributes 
about  4  percent  of  the  overall  support  for  computer  science 
research . 

The  ONR  is  responsible  for  all  the  basic  research  conducted 
by  the  Navy.  A  major  goal  of  the  ONR  Contract  Research  Program 
( CRP )  in  computer  science  is  to  fund  work  that  does  not  overlap 
with  DARPA's  program  and  bears  as  directly  as  possible  on  needs 
unique  to  the  Navy.  it  is  clear  that  ONR  must  try  to 
accomplish  its  goals  with  significantly  less  money  than  the 
Information  Science  Technology  office  at  DARPA.  Consequently, 
ONR  must  choose  its  problems  and  contractors  with  great  care. 

A  portion  of  ONR's  funds  is  earmarked  for  computer  science 
(CS)  research.  A  portion  of  the  CS  funds  is  administered  by 
the  CRP  of  ONR,  through  the  Computer  Science  Division  (CSD)  of 
the  Engineering  Sciences  Directorate,  while  another  portion  is 
allocated  directly  by  ONR  to  other  Navy  claimants,  such  as  the 
Nava],  Research  Laboratory  and  the  Naval  Commands.  Although  the 
ONR-CRP  funds  a  relatively  small  portion  of  all  computer 
science  research,  CRP  has  generally  chosen  excellent  projects, 
and  the  return  on  investment  has  been  high,  based  on  such 
indices  as  the  skills  and  prestige  of  the  principal 
investigators,  number  of  publications,  understood  significance 
of  the  findings,  and  impact  on  operational  naval  systems. 

The  Navy  expects  the  CRP  to  be  entrepreneurial,  i.e.,  to 
take  informed  risks  for  high  potential  payoff.  Through  CRP, 
the  Navy  hopes  to  identify  the  small  investments  in  computer 
technology  that  are  expected  to  have  the  greatest  impact.  The 
intent  is  to  discover  the  best  technology  and  begin  moving  Navy 
personnel  in  new  directions.  In  pursuit  of  this  goal,  CRP 
places  considerable  responsibility  on  the  program  managers. 

The  Panel  believes  that  this  is  the  primary  reason  for  CRP's 
success . 


TABLE  1  Estimated  Basic  Computer  Science  Research  Funding  by 
Federal  Agencies  for  Fiscal  Year  1986 _ 


Defense  Advanced  Research  Projects  Agency  (DARPA) 
National  Science  Foundation  (NSF) 

Navy 

Department  of  Energy 
Army 

Air  Force  Office  of  Scientific  Research 


>$190  million 
$  40  million 
$  10  million 
$  10  million 
$  S  million 
$  4  million 
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4.2.  ONR-CRP 1 s  Approach  to  Funding  Research 

In  October  1985,  the  ONR  computer  science  research  plan  was 
based  on  a  model  that  may  be  summarized  as  follows: 

1.  Slightly  under  half  of  ONR's  computer  science  research 
funds  are  allocated  to  the  research  activities  of  the  Naval 
laboratories  and  other  non-CRP  claimants.  Decisions  about 
appropriate  research  directions  are  made  by  the  laboratory 
managers  in  consultation  with  the  laboratory  scientists,  where 
the  work  is  performed  in  house,  and  by  program  managers,  where 
the  work  is  performed  external  to  the  Navy. 

2.  Slightly  over  half  of  ONR's  computer  science  research 
funds  are  allocated  to  small  ($50,000  per  year)  and 
medium-sized  ($150,000  per  year)  outside  contracts  administered 
through  CRP.  Decisions  among  alternative  contractors  are  made 
by  the  relevant  CRP  program  managers,  who  rely  in  turn  on 
informal  advice  from  leading  computer  scientists.  The 
technical  skills  and  prestige  of  the  principal  investigator 
carry  considerable  weight  in  the  decision  on  whether  to  fund  a 
proposal.  Program  managers  are  encouraged  to  have  an 
entrepreneurial  style;  they  have  the  authority  to  choose 
high-risk  projects  whose  payoff  would  be  very  high.  Program 
managers  are  responsible  for  tracking  the  progress  of  each  of 
their  projects  and  have  the  authority  to  withdraw  support  from 
projects  that  are  faltering  over  a  period  of  a  year  or  two. 

3.  New  research  directions  can  be  implemented  in  the  form 
of  accelerated  research  initiatives  (ARIs)  or  core  programs. 
ARIs  are  basic  research  programs  in  which  funding  is 
concentrated  for  a  limited  period  (nominally  5  years)  in  order 
to  accelerate  progress  in  the  field  of  research. 

It  is  useful  to  compare  the  above  ONR  model  with  those  used 
by  the  two  larqest  funding  agencies  for  computer  science 
research,  NSF  and  DARPA  (see  Table  1).  The  NSF  places 
considerable  weight  on  peer  review  of  proposals;  once  a 
proposal  is  accepted,  it  is  funded  in  the  form  of  a  grant. 
Follow-up  evaluation  is  used  for  large  projects,  such  as  the 
Experimental  CS  Program.  The  NSF  tries  to  cover  a  broad  range 
of  subdisciplines  and  does  so  with  many  awards. 

In  contrast,  DARPA  places  considerable  weight  on  the 
judgment  of  its  program  managers,  with  advice  from  leading 
researchers;  once  a  proposal  is  accepted,  it  is  funded  in  the 
form  of  a  contract.  DARPA  also  places  considerable  weight  on 
close  monitoring  of  the  progress  and  results  from  its  projects; 
its  primary  tool  is  regular  contractor  meetings  at  which  the 
set  of  contractors  constitutes  a  review  panel  for  the 
individuals  who  make  progress  reports.  DARPA  focuses  on  a  few 
selected  areas  and  makes  awards  in  large  sums  to  a  relatively 
small  group  of  contractors. 

The  ONR  CRP  follows  a  model  closer  to  that  of  DARPA.  The 
Panel  was  generally  pleased  with  this  approach.  It  appearea 


that  CRP  holds  contractor  meetings  less  frequently  than  DARPA 
does. 


4.3.  Current  CRP  Project; 


The  current  projects  receiving  ONR-CRP  funding  are  grouped  into 
categories  and  listed  in  Table  2.  The  Panel  listened  to  short 
descriptions  by  each  program  manager  of  each  task  in  his  area. 
The  Panel  did  not,  however,  attempt  a  technical  review  of  the 
individual  tasks  and  for  that  reason  will  not  offer  an 
evaluation  of  tasks  here.  The  Panel  does  believe  that  this  set 
of  projects  includes  a  number  whose  significance  has  already 
been  recognized  among  computer  scientists:  systolic  processing 
research  at  Carnegie-Mellon  University  (work  of  Kung); 
higher-order  languages  and  architectures  research  at  SRI 
International  (work  of  Coguen ) ;  semantic  theory  research  at 
Oxford  University  (work  of  Hoare);  visual  programming  research 
at  Brown  University  (work  of  van  Dam);  Poker  programming 
environment  research  at  the  University  of  Washington  (work  of 
Snyder);  program  visualization  research  at  SRI  International 
(work  of  Moriconi);  theorem  prover  research  at  the  University 
of  Texas  (work  of  Boyer  and  Moore);  analogy  system  research  at 
Carnegie-Mellon  University  (work  of  Carbonnel);  and  inferential 
programming  research  at  Carnegie-Mellon  University  (work  of 
Scott )  . 


TABLE  2  ON?  Contract  Research  Program  Project  Areas  in  Fiscal 
Year  1986 


Artificial  Intelligence 
Knowledge  acquisition 
Knowledge  representation 
Automated  reasoning 

$1.7 

million 

Robotics 

Reasoning 

Sensing 

Manipulation 

Mobility 

$1.2 

million 

Software  Engineering 

Software  foundations 
Science  of  software 

$1.5 

million 

Computer  Architecture 
Parallelism 

Distributed  systems 

$1.0 

million 
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In  listening  to  the  presentations  of  projects,  the  Panel 
noted  variations  in  presentation  style.  The  program  manager 
for  software  engineering  used  a  style  that  was  especially 
helpful:  with  each  progect  he  stated  an  hypothesis,  ana 

objective,  and  an  approach.  The  hypotheses  were  sufficiently 
specific  that  reviewers  would  be  able  to  assess  the  success  of 
the  project  at  meeting  its  goals.  Each  project  had  measurable 
milestones  stated  for  it.  The  Panel  understands  that  this 
information  is  derived  from  standard  forms  used  by  ONR  to 
summarize  all  its  projects.  The  Panel  encouraged  all  the 
program  managers  of  CRP  to  use  this  style  when  making 
presentations  to  outside  review  bodies. 

During  the  presentations  of  the  computer  science  programs, 
several  items  caught  the  Panel's  attention.  The  Panel  notes 
them  here  and  encourages  the  program  managers  to  clarify  them 
in  case  they  come  up  again  in  the  future: 

1.  In  the  artificial  intelligence  area,  it  appears  that 
progects  to  analyze  and  refine  basic  techniques  are  being 
gudged  with  metrics  that  are  more  suited  to  exploratory 
progects.  Some  of  the  claims  are  stated  nonquanti tatively , 
making  it  difficult  to  assess  progress.  Examples  are  "changing 
to  a  better  programming  paraoigm"  and  "improving  the 
reliability  of  programs."  The  Panel  understands  that  AI 
researchers  have  addressea  some  of  the  most  difficult  progects 
in  computer  science  and  that  the  nature  of  these  problems 
frequently  makes  quantitative  analysis  difficult,  but  the  Panel 
nonetheless  encourages  program  managers  to  seek  verifiable 
claims  as  they  do  in  robotics,  software  engineering,  ana 
architecture.  Examples  of  questions  that  researchers  should  be 
encouraged  to  explore  are:  How  powerful  is  the  method?  What 
size  problems  can  it  handle?  What  are  the  ultimate  limitations 
of  the  approach?  What  resources  might  be  required  for 
realistic  tasks?  The  obgective  is  to  inject  more 
quantification  and  evaluation  into  research  to  guiae  the 
discovery  process  and  to  enhance  the  ability  to  refine  and 
improve  the  field. 

2.  Several  of  the  theoretical  computer  science  progects 
supported  by  the  CRP  listed  vague,  unspecific,  or  generic 
goals,  such  as  "new  approaches  to  algorithms,"  that  obscured 
the  relevance  to  the  Navy's  long-term  objectives.  The  Panel 
encourages  program  managers  to  seek  clearer  statements  of  the 
goals  of  the  theory  progects  and  the  potential  benefits  to  the 
Navy . 

3.  There  are  several  investigators  who  have  been  under 
contract  to  the  Navy  for  lengthy  periods,  e.g.,  more  than  7 
years.  It  appears  to  the  Panel  that  the  benefits  of  their  work 
were  not  clearly  articulated.  The  Panel  encourages  the  program 
managers  to  review  such  progects  and  state  their  benefits  to 
the  Navy  more  clearly. 


14 


A  *  ■*. 

'•yyw 

y  -\V.  1 

.  • 


v'-.-VV'J 


ivS-Sv 

s-S' ££"3 


•V*  .  v  - .  ■ 


•  /'  s  • 

•  »  »  •  * 

■ V 

‘■•TV  V-.l 

r?  ~  ■*  r  ■ 


»  >  ■» 


i  m.  *  ,r\  W1  d  «Tw  “Tw  A.VvTT^'  ^  /■  .  #  -  X  ^.iT.^.riirj  -ji  v  r  »  -»  t»  ■  -_* 


•  - 


4.  There  appears  to  be  significant  overlap  between 
software  engineering  research  in  cooperating  sequential 
processes  (CSP)  and  artificial  intelligence  work  in  distributed 
computing.  While  the  Panel  has  no  objection  to  redundancy  in 
research  programs,  it  appears  that  the  program  managers  were 
not  encouraging  the  researchers  to  exchange  ideas. 

The  Panel  believes  that  program  managers  should  make  more 
systematic  use  of  techniques  to  obtain  advice  and  evaluation  of 
projects.  The  Panel  suggests  four  mechanisms  that  can  be 
employed  without  compromising  the  successful  style  of  CRP: 

1.  Follow  DARPA's  lead  in  holding  contractor's  workshops 
(at  least  one  annually).  Request  each  contractor  to  present  a 
progress  report  and  use  an  open  discussion  to  ferret  out 
weaknesses  and  identify  strengths.  With  this  technique,  the 
contractors  themselves  review  each  other's  projects;  the 
workshops  give  important  feedback  to  the  program  managers  for 
evaluating  progress.  For  added  benefit,  such  workshops  might 
also  include  non contractors  active  in  the  field. 

2.  Follow  NSF's  lead  by  establishing  a  formal  advisory 
panel  that  meets  with  program  managers  once  or  twice  a  year  for 
program  review  and  general  guidance. 

3.  Make  the  ONR  standard  formats  for  describing  projects 
more  prominent.  These  formats  request  the  investigator  to 
state  a  vision  of  the  expected  long-term  accomplishments  of  the 
project.  They  also  request  the  investigator  to  state  specific 
milestones  that  can  be  expected  along  the  way.  The  Panel 
strongly  encourages  program  managers  to  seek  quantitative 
milestone  statements,  which  observers  can  evaluate  later  to  see 
whether  they  were  achieved.  (Examples  of  quantitative 
statements  are;  "Exhibit  a  prototype  theorem  prover  with  the 
following  properties..."  and  "Develop  lower  bounds  on  the  time 
to  solve  problems  of  X  type  on  a  mesh  computer.") 

4.  Publicize  important  results,  e.g.,  in  an  ONR  newsletter 
or  by  a  new  publication  similar  to  the  National  Research 
Council's  News  Report. 

The  first  mechanism  in  the  list  above  would  help  program 
managers  to  obtain  regular  and  systematic  information  to  assess 
which  projects  are  doing  well  or  poorly.  The  second  mechanism 
would  help  program  managers  to  evaluate  their  own  performance 
in  selecting  projects  and  would  give  regular  guidance  about 
long-term  research  directions.  The  third  mechanism  would  help 
observers  assess  the  purposes  and  accomplishments  of  each 
project  and  each  program  area.  The  fourth  would  bring  ONR 
accomplishments  into  wider  prominence. 

Some  of  the  program  managers  discussed  their  feelings  about 
project  size.  Some  felt  that  theory  projects  could  be 
adequately  supported  with  as  little  as  $50,000  per  year  but 
that  a  significant  system-development  project  might  require 
approximately  $300,000  per  year.  The  Panel  believes  that  the 
figure  for  a  system-development  project  may  be  low,  given  the 
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costs  of  equipment  and  support  personnel.  The  Panel  would 
recommend  a  smaller  number  of  well-done  projects  over  a  larger 
number  of  underfunded  projects. 

4.4  Summary  of  Findings  and  Recommendations 

The  Panel  was  generally  pleased  with  what  was  presented  and 
with  Navy  personnel.  The  Panel  believes  that,  on  the  whole, 
the  various  computer  science  program  managers  have  done  a 
first-rate  job  in  identifying  and  supporting  good  projects. 

From  its  broad  science  experience,  the  Panel  judged  that  the 
CRP ' s  computer  science  research  program  includes  some  of  the 
best  computer  science  projects  in  the  country;  the  Panel  cited 
ten  examples.  The  Navy,  and  especially  ONR's  CRP,  appears  to 
have  achieved  an  excellent  return  on  investment  for  the 
research  dollar.  The  strengths  of  the  CRP  program,  especially, 
are  its  flexibility,  its  entrepreneurial  style,  and  its  success 
as  a  computer  technology  showcase  for  the  rest  of  the  Navy. 

The  CRP's  basic  approach  to  managing  its  program  is 
successful  and  should  be  preserved.  The  main  features  are 
considerable  responsibility  given  to  program  managers, 
encouragement  of  informed  risk-taking,  sticking  with 
researchers  of  exceptional  talent  or  expertise,  and  allocating 
a  portion  of  the  funds  for  accelerated  research  initiatives. 

CRP  and  other  claimants  on  ONR  funds,  such  as  NRL  and  the 
Systems  Commands,  are  encouraged  to  review  their  mechanisms  for 
obtaining  advice  and  evaluating  programs,  looking  especially  at 
contractor  meetings,  an  advisory  panel,  standard  reporting 
formats,  and  improved  publicity  for  results. 

One  of  the  strengths  of  the  program  is  also  a  potential 
weakness:  the  success  of  the  program  depends  strongly  on  the 

judgments  of  a  few  people.  The  Panel  has  suggested  mechanisms 
that  would  get  more  advice  for  program  managers  without 
restricting  their  judgments  or  interfering  with  the  other 
strengths  of  the  program.  The  Panel  emphasizes  that  there  is 
no  serious  problem  now. 

Some  areas  of  the  technical  program  should  be  reviewed, 
including  (1)  nonquantitative  claims  made  by  some 
artificial-intelligence  contractors,  (2)  relevance  of  theory 
areas  to  the  long-term  goals  of  the  Navy,  (3)  flow  of  benefits 
from  investigators  who  have  received  contracts  for  extended 
periods  (over  7  years),  and  (4)  overlaps  between  distributed 
computation  in  computer  architecture  and  artificial 
intelligence.  The  Panel  considers  none  of  these  matters 
serious . 
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The  Panel  identified  five  areas  of  research  that  are  critical 
to  future  naval  systems  but  were  not  well  covered  by  the 
computer  science  research  efforts  presented  to  the  Panel  in 
October  1985.  All  these  areas  require  substantial  basic 
research  before  practical  results  can  be  expected. 

1.  Real-Time  Systems.  The  area  of  real-time  systems  is  an 
engineering  discipline  looking  for  a  scientific  basis.  The 
technology  base  for  real-time  systems  is  changing  drastically. 
Because  they  are  intrinsically  highly  parallel  in  structure, 
real-time  systems  are  rapidly  impacted  by  parallel 
architectures.  Microelectronics  is  creating  a  new  generation 
of  sensors  capabilities  for  dedicated,  front-eno  processing  of 
sensor  data.  Artificial  intelligence  offers  the  potential  for 
increasing  the  capabilities  of  control  systems.  Distributed 
systems  and  high-speed  networks  are  offering  new  ways  to 
connect  sensors,  activators,  processors,  and  other  system 
components.  A  new  ONR  research  initiative  could  seek  to 
establish  a  scientific  basis  for  a  new  generation  of  real-time 
systems  that  exploit  these  new  potentials.  Each  of  these 
technology  areas  has  its  own  claims  for  attention  and  funding; 
the  critical  point  is  the  need  to  integrate  these  technologies 
into  coherent  real-time  systems. 

2.  System  Structure  Concepts.  Most  Naval  computer  systems 
include  large,  complex  software  systems  that  are  the  results  of 
thousands  of  hours  of  human  effort.  Software  engineering  seems 
to  be  falling  behind  the  effects  of  new  technology,  such  as 
parallel  and  distributed  computing  systems.  For  example, 
environments  for  developing  and  debugging  parallel  and 
distributed  programs  have  yet  to  be  developed;  multimedia  mail 
and  real-time  conferencing  have  not  been  integrated  into  the 
programming  environment;  interface  systems  cannot  interact  with 
users  at  the  level  of  abstraction  of  the  application; 
distributed  groups  of  people  cannot  collaborate  effectively 
over  the  networks;  software  cannot  easily  be  reused.  There  is 
a  need  to  establish  and  evaluate  new  system  development  and 
structure  concepts  that  deal  with  these  new  realities.  A  major 
need  appears  to  be  better  integration  of  elements  of  the 
development  process  from  application  through  hardware. 

3.  parallel  Computing.  This  area  is  important  to  Areas  1 
and  2,  above.  It  may  be  subdivided  into  two  parts.  First, 
future  computer  hardware  will  rely  significantly  on  parallel 
processing  to  achieve  greater  computational  power.  However, 
little  is  known  about  parallel  programming  languages  or  hew  to 
write  parallel  software,  so  research  should  be  directed  to  the 
effective  use  of  current  and  future  parallel  machines.  This 
research  should  encourage  strong  interaction  between  potential 
users  and  architects  of  parallel  machines.  The  second  subarea, 
distributed  problem  solving,  involves  concepts  for  coordinating 
multiple  processors  of  different  types,  especially  when  the 
processors  and  communications  are  likely  to  fail.  This  is 
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obviously  important  in  battlefield  situations  that  may  include 
substantial  numbers  of  distinct  computer  systems. 

4.  Formalization  of  Expert  Systems.  The  Panel  heard 
statements  that  considerable  reliance  may  be  placed  on  expert 
systems  in  future  Naval  systems.  Most  expert  systems  today  are 
based  on  informal  heuristics  and  have  uncertain  limits  of 
competence.  The  Panel  believes  that  expert  systems  can  be 
pushed  much  harder  in  the  direction  of  greater  reliability  and 
competence.  If  the  problem-solving  model  underlying  the 
heuristic  knowledge  in  an  expert  system  could  be  represented 
formally,  it  would  be  possible  to  predict  the  behavior  of  the 
system  in  untested  cases  and,  more  generally,  to  understand  the 
limits  and  capabilities  of  a  given  expert  system.  This  would 
be  especially  important  where  expert  systems  are  relied  on  to 
assist  in  time-critical  decision-making. 

5.  Representation  of  Physical  Objects.  Naval  systems  rely 
on  large  numbers  of  complex  physical  objects  and  parts.  One 
special  type  of  reliance  is  on  the  replaceability  of  spare 
parts.  Replaceability  of  parts,  as  well  as  their  initial 
manufacture,  would  be  significantly  improved  by  representing 
the  parts  in  computer-usable  form  from  their  inception.  This 
would  include  not  only  the  parts,  their  attributes,  and  their 
interconnections  but  also  the  decisions  that  went  into  their 
initial  designs.  Spare  pares  could  be  manufactured  easily  as 
needed  by  retrieving  their  descriptions  from  a  database  and 
feeding  them  to  a  programmable  production  facility. 
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APPENDIX  A 


Panel  on  Assessment  of  Computer  Science  Activities  of  the 
Office  of  Naval  Research  (ONR) 

Naval  Studies  Board 

Commission  on  Physical  Sciences,  Mathematics,  ana  Resources  (CPSMR) 

National  Research  Council 

Tuesday  and  Wednesday 
S-9  October  1985 
Room  453,  Joseph  Henry  Building 
2122  Pennsylvania  Avenue,  N.W. 

Washington,  D.C.  20418 


Agenda 


Tuesday,  8  October  1985 

0900  Welcome  and  Introductions  Peter  J.  Denning, 

Chairman 

0915  Introductory  Statement  and  Lawrence  E.  McCray, 

Discussion  of  "Potential  Sources  Associate  Executive 
of  Bias"  and  Conflicts  of  Director,  CPSMR 

Interest 

0930  Discussion  Concerning  the  Nature  Ronald  N.  Kostoff, 

of  the  Assessment  Director,  Technical 

Assessment,  ONR 

0945  Issues  to  be  Addressed  Arthur  M.  Diness 

Associate  Director/ 
Research  (Engineering 
Science ) ,  ONR 

1015  BREAK 

1030  The  Navy's  Computer  Science  Bruce  Wald,  NRL 

Program;  User  Requirements: 

— A  Naval  Research  Laboratory 
Perspective 

1100  — A  Naval  Air  Systems  Command  S.  A.  Gerhard  Heicke, 

Perspective  Nav  Air 

1130  Overview  of  ONR's  Computer  Alan  L.  Meyrowitz, 

Science  Program/Art if ic)al  Information  Sciences 

Intelligence  Division,  ONR 

1215  Discussion 


1230 


LUNCH:  Committee  Dining  Room  1 


1345 

ONR  Programs  (Continued): 
--Robotics 

Alan  L.  Meyrowitz 

1415 

--Software  Engineering 

Robert  B.  Grafton, 
Information  Sciences 
Division,  ONR 

1515 

BREAK 

15  30 

--Computer  Architecture 

David  W.  Mizell, 
Information  Sciences 
Division,  ONR 

1630 

Discussion 

1700 

Recess  until  0630 

Wednesday,  9  October  1985 

0830 

Other  Navy  Computer  Science 
Research  Programs: 

--Naval  Research  Laboratory 

Randall  Schumaker, 
Director,  Navy 

Center  for  Applied 
Research  in 
Artificial 
Intelligence,  NRL 

0930 

--Naval  Systems  Commands: 

Naval  Air  Systems  Command 

6.1  Research 

James  G.  Smith, 

Nav  Air 

1000 

BREAK 

1015 

--Space  and  Naval  Warfare 

Systems  Command 

Steve  Sachs,  John 
Machado,  ana  John 
Pucci,  spawars 

1115 

Other  6.1  Computer  Science 
Research  Programs  of  the  Navy 

1200 

General  Discussion 

1230 

LUNCH:  Committee  Dining  Room  1 

1345 
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